On The Direct Maximization of Quadratic Weighted Kappa

In recent years, quadratic weighted kappa has been growing in popularity in the machine learning community as an evaluation metric in domains where the target labels to be predicted are drawn from integer ratings, usually obtained from human experts. For example, it was the metric of choice in several recent, high profile machine learning contests hosted on Kaggle : www.kaggle.com/c/asap-aes , www.kaggle.com/c/asap-sas , www.kaggle.com/c/diabeticretinopathy-detection . Yet, little is understood about the nature of this metric, its underlying mathematical properties, where it fits among other common evaluation metrics such as mean squared error (MSE) and correlation, or if it can be optimized analytically, and if so, how. Much of this is due to the cumbersome way that this metric is commonly defined. In this paper we first derive an equivalent but much simpler, and more useful, definition for quadratic weighted kappa, and then employ this alternate form to address the above issues. 1. Preliminaries Although first developed in the statistical community as a measure of inter-rater agreement, κ has more recently become a popular performance metric in supervised machine learning, specifically in situations where the target (dependent) variable y is a discrete, interval variable (usually drawn from non-negative integers) such as is common in most human rating scales (e.g.“on a scale from 1 to 10”). This differs from the ordinal regression setting, where there only exists an ordering over labels, but no intrinsic or constant length interval between them. Some have argued that the use of quadratic weighted kappa as a metric in the domain of human ratings imposes the erroneous assumption of “equal intervals” where there should be none (how this assumption is expressed in the metric itself will be made clear in the following section). For example, when rating student essays on a scale from 1 to 5, the difference between a 1 and a 2 may not be equal to the difference between a 4 and a 5. While this may or may not be true in certain cases, we will not be concerned with that here. 1.1. Standard Definition Quadratic weighted kappa, which we write κ to distinguish from linear weighted kappa, was originally developed as a measure of inter-rater agreement. In this scenario, there are two raters, A and B, each associated with a vector of n integer ratings a,b ∈ Ln×1 where L = {1, 2, · · · , `} is a finite set of ` possible values. We seek to quantify the level of agreement between a and b. In order to compute κ(a,b), it is customary to start by computing frequency tables. The observed confusion matrix U = (ui,j) ∈ N`×` is first computed as: